Gill’s Framework

• Brian Gill’s (2022) paper, What should the future of educational accountability look like?, introduces a useful framework for indicators used in accountability systems.
• Three Core Components of Gill’s Framework
  • Processes: The activities, instructional practices, and organizational elements that schools implement to support student learning.
  • Impacts: The measurable effects of processes on student learning and development, often captured through growth metrics or value-added models.
  • Outcomes: The long-term results of education, such as achievement/proficiency levels, graduation rates, and other indicators of student success.
• Gill argues that for indicators to drive meaningful improvement, they must be:
  • Valid: Accurately measure what they claim to measure without systematic bias.
  • Reliable: Stable and consistent over time to avoid misdiagnosis or loss of credibility.
  • Robust: Resistant to unintended consequences and manipulation while capturing critical dimensions of educational quality.

Gill’s Infographic

Coherent Components

Why Coherence is Critical?

Performance Management versus Accountability

• Performance management is a comprehensive, ongoing effort to improve an organization’s effectiveness, efficiency, and outcomes.
• Accountability is just one component of performance management—focused on measuring, reporting, and ensuring responsibility for outcomes.
• While performance management emphasizes continuous improvement, accountability primarily evaluates past performance.
• Effective performance management uses accountability as a tool, but also incorporates support, innovation, and strategic planning.
• Accountability without performance management can lead to compliance-driven behaviors rather than meaningful improvement.
• A coherent system integrates accountability within a broader performance management framework, ensuring data is used to drive improvement rather than just to judge performance.

The Reign of Proficiency

• During the last week of January, National Assessment of Educational Progress (NAEP) results were released.
• If you read beyond the headlines, an implicit message about communicating the performance of our education system is crystal clear: Proficiency reigns supreme.
• Since at least the NCLB era, education policy has been fixated on proficiency levels and their trends (i.e., achievement/attainment).
• Call it what you will —- proficiency, mastery, college and career readiness -— student attainment is the primary measure of educational performance.
• Proficiency is King! And it should be!
• Why? Claiming an education system is “good” when only a small fraction of students are proficient is absurd.
• But here’s the catch: Proficiency and proficiency trends are the wrong tools for the effective performance management necessary to improve an education system.
• Death to the King! Long Live the King!

Death to Proficiency!

• Since at least the No Child Left Behind (NCLB) era, attainment has had its detractors.
• Rightly, these critics have pointed out that using attainment for accountability purposes (e.g., school ratings) is problematic due to selection bias – students are not randomly assigned to schools.
• The solution to this problem has been to incorporate growth or value-added into the accountability system.
• Over the last 15 years, one of the primary foci of accountability system design has been weighting attainment and growth components to create more “valid” accountability indicator systems.
  
• The result, I would argue, is a design process that treats these two components like food items at a buffet.
• And today several efforts to create more valid accountability systems promote adding more components.
• The result of this process of adding more components, I contend, is incoherence in the design of accountability systems.

Attainment & Learning

• And as indicators used for accountability have increased, a significant part of the ensuing incoherence has been the imprecise, inconsistent, and incorrect use of terminology.
• Just considering results from large-scale assessments, the number of different terms is daunting:
  • Achievement, proficiency, attainment, status, and mastery are several terms frequently encountered to discuss what I refer to as attainment, a characterization of what a student or groups of students know and can do at a point in time.
  • Growth, value-added, learning, SGPs, value-tables, content-referenced growth are several terms used to characterize student growth of a student or group of students.
  • Improvement is a term that seems to be used to describe a beneficial change (either for a student or a group of students).
  • Acceleration is a term that came to prominence during the pandemic to motivate goals to correct learning loss by accelerating student learning.
• Having attended hundreds of meetings and presentations over the last decade, I can’t count the number of times I’ve heard several of these terms used in ways that have led me to believe there is a fundamental lack of understanding about how large scale assessment results should be used.

Coherence in Accountability System Design

• There is no shortage of criticism of current accountability systems.
• A substantial part of many criticisms is the over-reliance on the results from large-scale assessments.
• I would argue that the root of the problem isn’t the over-reliance on the results from large-scale assessments, but rather the incoherent way in which results from large-scale assessments are utilized.
• Accountability systems are the result of policy initiatives: A coherent accountability system ideally connects the larger policy goals of the education system (increased rates of proficiency and the closing of achievement gaps) with the processes associated with achieving (or not achieving) those goals.
• Even with coherence, there is no guarantee that the education system will achieve the policy goals.
• But without coherence, I contend, accountability initiatives intended to improve the education system are doomed to fail.

Coherence

• The term coherence is not frequently used in discussions of accountability systems and when it is, it is often not clearly defined.
• What is coherence with respect to accountability systems?
• What is incoherent about current accountability systems?
• Why is coherence important?
• How can coherence be instantiated in accountability systems?
• What benefits does coherence provide?

–